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Abstract 

Conducted in conjunction with an institute on open textbook adaptation, this study compares 
textbook evaluations from practicing K-12 classroom teachers (n = 30) on three different types of 
textbooks utilized in their contexts: copyright-restricted, open, and open/adapted. Copyright- 
restricted textbooks consisted of those textbooks already in use by the teachers in their 
classrooms prior to the institute, open textbooks included alternatives from CK-12 and OpenStax, 
and open/adapted consisted of open textbooks that the teachers devoted time to adapting to their 
individual needs. Results indicate that open/adapted textbooks were evaluated as having the 
highest quality and that open textbooks were of higher quality than copyright-restricted 
textbooks. Though some factors of quality might be influenced by cost differences (e.g., timeliness 
and the ability to adopt updated textbooks), results reveal that open and open/adapted textbooks 
may do a better job of meeting the needs of K-12 teachers in a variety of ways that may not be 
captured through traditional approaches to quality assurance. This study marks an early step in 
exploring the quality of K-12 open educational resources (OER) and the use of practicing teachers 
as authentic evaluators of textbooks for their local contexts. 

Keywords: open education; OER; K-12; open textbooks; professional development; OER 
quality; quality assurance 
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Introduction 

Open educational resources (OER) are “any type of educational materials that are in the public 
domain or introduced with an open license” (UNESCO, n.d., para. 1) or are “teaching, learning, 
and research resources that reside in the public domain or have been released under an 
intellectual property license that permits their free use and re-purposing by others” (William and 
Flora Hewlett Foundation, n.d., para. 2). As such, OER differ from traditional copyright-restricted 
educational materials in that “anyone can legally and freely copy, use, adapt and re-share them” 
(UNESCO, n.d., para. 1) provided that those who use these materials adhere to specific guidelines 
of use that may vary from resource to resource (e.g., attributing the author, non-commercial 
redistribution). OER have gained much attention in recent years as a means to increase access to 
quality educational materials, especially in underserved parts of the world (Atkins, Brown, & 
Hammond, 2007; Baraniuk, 2007; D’Antoni, 2008; Mtebe & Raisamo, 2014; Prasad & Usagawa, 
2014), to drive down educational costs (Bliss, Hilton, Wiley, & Thanos, 2013; Wiley & Green, 
2012; Wiley, Hilton, Ellington, & Hall, 2012), and to support valuable collaborative practices 
between educators and students (Brown & Adler, 2008; Petrides, Jimes, Middleton-Detzner, 
Walling, & Weiss, 2011). Furthermore, though adoption of OER began largely in higher education, 
we are now seeing its influences trickle down to K-12 institutions and public education in the U.S. 
(Bliss & Patrick, 2013; K12 OER Collaborative, 2014; Usdan & Gottheimer, 2012). 

Like any educational materials, OER come in a variety of forms and are shared via a variety of 
media. OER repositories like the Digital Public Library of America (http://dp.la), OER Commons 
(http://oercommons.org), Merlot (http://merlot.org), and many others have been created and 
curated to connect educators to digital OER in a meaningful manner via hosting, rating, 
categorizing, tagging, and otherwise sharing these resources (Atenas & Havemann, 2013). In a 
more traditional manner, however, OER have also been explored as replacements for pre-digital 
media in the form of OER textbooks, wherein the common textbook structure is used as a means 
for providing OER to schools and higher education institutions in a familiar manner (Baker, 
Thierstein, Fletcher, Kaur, & Emmons, 2009). Such textbooks can be freely shared and 
distributed digitally as e-books and may also be freely printed as hardcopy books without 
licensing fees or seeking author permission (Hilton & Laman, 2012). Projects such as OpenStax 
College (n.d.) and CK-12 (n.d.) provide OER in this textbook format, and because these textbooks 
are openly licensed, educators and students are free to download and utilize them as-is or to 
adapt them to their individual needs. This approach to content licensing and curation is of special 
interest to those interested in online and blended learning, since OER may be freely distributed 
via electronic means, and OER adoption for online instruction can further drive down cost in this 
medium while simultaneously improving access for students (e.g., no logins are necessary to 
access content) and ease of implementation by designers (e.g., content may be readily 
incorporated into any online instructional system without additional permissions). 
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Though the relative low cost of OER is undeniably tempting to educators, students, and those 
financing education, cost is not the only factor driving curricular adoptions. For instance, a recent 
survey of higher education faculty found that the cost of a textbook is of relatively low importance 
to faculty members when selecting which materials to adopt as compared to other content quality 
factors, such as “proven efficacy” or trustworthiness of the creator (Allen & Seaman, 2014; cf. 
Clements & Pawlowski, 2011). Despite the fact that higher education students may view open 
textbooks as a comparative or even better alternative to copyright-restricted textbooks (Bliss, 
Hilton, Wiley, & Thanos, 2013), their instructors and others making curricular decisions must 
sign off on the content that is being provided. In this situation, the traditional publication 
mechanism, which relies upon peer review and authorship by content experts, is used as a means 
for verifying quality, and since OER does not generally undergo a similar process, educational 
institutions may perceive them as lacking a necessary stamp of approval that may not overcome 
the cost incentive (cf. Prasad & Usagawa, 2014). 

Quality OER undoubtedly exist, and research into crowd-generated content has been promising 
in showing that even platforms as open as Wikipedia can produce materials that live up to 
reasonable standards of quality, at least in terms of traditional measures such as content accuracy 
(Giles, 2005). On the other hand, poor-quality OER also undoubtedly exist, and though 
stakeholders recognize that quality control is an essential priority for OER adoption (D’Antoni, 
2008), it is currently unclear how to best ensure quality in lieu of a traditional publication 
process, and the perceived lack of quality that results from this situation marks a major barrier for 
OER adoption (de Hart, Chetty, & Archer, 2015; Kimmons, in press). Furthermore, though some 
valuable studies have begun establishing student outcome measures to determine the effects of 
particular OER use on student achievement (Robinson, Fischer, Wiley, & Hilton, 2014; Wiley, 
Hilton, Ellington, & Hall, 2012; cf. Walker, 2007), these studies need to be supplemented with 
research that considers the quality of the resources themselves in order to support decision¬ 
making and to control for implementation factors that might influence results. 

Some OER undergo peer review (e.g., OpenStax) and are authored by content experts (e.g., CK- 
12), but such examples are not the norm, and generally speaking, “[OER] quality control is in the 
hands of learners and teachers ... instead of instructional [or content] experts” (Dinevski, 2008, p. 
121), and quality assurance in OER is seen as “primarily the responsibility of ... education 
institutions” (UNESCO & COL, 2011, p. 13). This represents a paradigmatic shift in quality control 
processes, wherein evaluations of quality must be conducted by the educators and institutions 
that will be using the products and must also rely upon the suitability of the products themselves 
rather than upon the processes (e.g., peer review) that produced them (cf. Wiley, 2013). Though 
some work has been done to begin systematizing the evaluation of OER repositories (e.g., Atenas 
& Havemann, 2013; Atenas, Havemann, & Priego, 2014; OECD, 2007), research on local 
evaluations of OER by educators and educational institutions is a largely uncharted and daunting 
territory that requires intense contextualization and a knowledge of how such institutions operate 
(Clements & Pawlowski, 2011; Pawlowski, 2007). 
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Fledgling approaches to quality assurance are especially problematic for K-12 institutions, for 
whom cost is a much greater incentive in driving a shift to OER, and for whom time constraints 
for evaluation might be more severe than for university faculty (Kimmons, in press). Though 
problematic, this shift may also represent a much-needed, positive opportunity for K-12 
institutions to shape the quality control process in a manner that will be beneficial for teachers 
and students. The reason for this possibility is that traditional quality assurance mechanisms may 
not effectively respond to the authentic realities of K-12 institutions, and the quality of a resource 
in K-12 institutions involves much more than mere content accuracy and proper formatting 
(Carey & Hanley, 2007). 

Consider the factors of cost and content accuracy and how this differs between K-12 and higher 
education institutions. The cost of a resource in higher education may not be correlated to 
accuracy, because students may be directed to purchase new versions of the assigned textbook 
each semester, and instructors may guide students to purchase these resources without 
consideration of cost (Allen & Seaman, 2014). This means that university students will generally 
be expected to have up-to-date materials even if these materials are expensive. In public K-12 
institutions in the U.S., on the other hand, textbook purchasing works very differently. K-12 
institutions purchase textbooks on behalf of their students and operate on textbook adoption 
cycles that are dependent upon budget planning and availability of resources. Textbooks are 
purchased for a lengthy (sometimes non-predetermined) cycle, which in many schools may 
extend to ten years or longer (FCC, 2012). This means that K-12 teachers maybe required to teach 
out of textbooks that are outdated or inaccurate for several years due to the high cost of replacing 
the resource. This situation is further exacerbated when states adopt new standards, such as the 
Common Core State Standards (CCSSI, n.d.) or Next Generation Science Standards (n.d.), which 
require large-scale curricular redesigns. Schools that cannot afford to replace all of their 
textbooks simultaneously in response to such shifts are thereby required to teach out of materials 
that no longer address mandated standards. Thus, in K-12, quality is a function of cost, because 
cost drives the ability of teachers to gain access to updated materials that are aligned to current 
standards. 

Since OER are freely available, however, they can be adopted at any time and can thereby 
accelerate or altogether bypass the traditional adoption cycle. This allows teachers to acquire up- 
to-date, accurate, standards-aligned materials in a timely manner, but peer review and traditional 
quality assurance measures ignore this factor. Resource quality in K-12 is additionally influenced 
by a number of other factors that may differ from higher education, including alignment with 
standards, content conciseness, content reading level, content engagement, media accessibility, 
and supplementary materials, such as learning activities, rubrics, and lesson plans (Carey & 
Hanley, 2007; Mtebe & Raisamo, 2014). The idea of the textbook as an information source, on the 
other hand, seems to be highly influenced by higher education, and for this reason textbooks tend 
to be treated as content sources that are created and reviewed by content experts rather than 
educators. Most textbooks used either at K-12 or higher education levels are written by college 
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professors or other professionals with PhDs in the content area, not classroom teachers or 
pedagogues. This means that though the content may be accurate and up-to-date when it is 
written, there may be a variety of limitations in its design, including the following: 

1 ) It may not be written at a reading level appropriate for classroom students; 

2 ) It may not be organized in a manner that aligns with state standards or with 
scaffolded learning in the subject area; 

3 ) It may not include supplementary resources that can be useful for students and 
teachers, such as meaningful practices, quizzes, lesson plans, classroom activities, 
rubrics, and so forth; 

4 ) It may not be written in a manner that allows teachers to easily integrate the material 
into diverse classrooms in response to cultural, community, or geographic 
considerations or to individual student needs (Kimmons, in press). 

This means that the nature of the textbook medium as it is traditionally conceived yields the 
creation of resources that may not reflect indicators of quality that are important to K-12 teachers. 
Though accurate, up-to-date content is an essential element of quality, K-12 teachers need 
resources that can be differentiated for students’ diverse learning needs and that can be easily 
integrated into institutional and state requirements of teachers, through meaningful standards 
alignment and the provision of appropriate supplementary materials. The reality of the situation, 
however, is that textbook companies approach K-12 educational institutions as a general market 
wherein a textbook may be developed for a large state such as Texas or California and then 
minimally adjusted to theoretically meet the standards adopted in other states as well. This 
means that rather than being designed for local-, teacher-, or student-level needs, textbooks are 
designed for state-level needs and may not even be designed specifically for the states in which 
they are adopted. Furthermore, since students, teachers, and sometimes local K-12 institutions 
are not the entities making the decision on adoption, it is questionable whether their voices are 
heard in the adoption process and whether their needs are being met by the materials being 
provided to them. 

For these reasons, a paradigmatic shift in quality assurance for K-12 textbooks is not only feasible 
but may be long overdue, and OER represents an opportunity to make this shift. Quality cannot 
exist without an intended context, and the quality of OER, as with any educational resource, is 
connected with the contexts in which they will be used (Clements & Pawlowski, 2011). Because 
individual educators are the best experts on their local contexts, it seems to make sense for these 
educators to be the evaluators of the resources for their contexts. This study seeks to explore how 
a group of practicing educators evaluate existing textbooks from CK-12 and OpenStax in 
comparison to their currently-used, copyright-restricted alternatives. 
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Breaking away from the limitations of the copyright-restricted paradigm, however, this study also 
attempts to take the next step to consider how the process of adaptation, rather than mere 
adoption, of OER helps to improve the quality of these resources for practicing educators. Like 
copyright-restricted materials, OER may be adopted without any form of adaptation (Duncan, 
2009; Hilton & Wiley, 2012; Prasad & Usagawa, 2014), but they can also be remixed, reorganized, 
revised, updated, reworded, and otherwise adjusted in response to educator needs (Tonks, 
Weston, Wiley, & Barbour, 2013). In a perfect world, we can envision K-12 teachers being the 
content experts in their classrooms and having the resources and flexibility necessary to adopt, 
create, revise, and adapt educational materials to their students’ needs. However, within the 
existing paradigm, teachers are not only typically required to utilize curricula that they have no 
voice in choosing, but because of the copyright-restricted nature of the materials, they also cannot 
update, adjust, reword, or otherwise alter the materials that they have been given. Teachers must 
make their students malleable to the content rather than make the content malleable to their 
students. 

This reveals a significant difficulty with current approaches to teaching within a copyright- 
restricted content paradigm that is less than ideal and that restricts teachers from making full use 
of the limited resources that they have. However, OER may offer a means by which teachers and 
K-12 institutions can gain the ability to more meaningfully differentiate instruction for their 
students and to share valuable classroom resources with those beyond their local contexts. Thus, 
in this study we will also guide teachers in the adaptation of their own textbooks to meet their 
diverse classroom needs and use this as a basis of comparison to determine the relative quality of 
adapted OER to non-adapted OER and to copyright-restricted alternatives. 

Methods 

This study attempts to determine whether open educational resources (OER) and specifically 
open textbooks can be as high quality as copyright-restricted alternatives for K-12 educators and 
also to determine whether the ability to adapt these textbooks to local or classroom needs makes 
any difference in their quality. We use practicing K-12 middle- and high-school teachers in this 
study as evaluators in place of traditional content-expert peer reviewers. We do this for a number 
of reasons, but primarily because we recognize that teachers are the only professionals qualified 
to recognize the needs of their students and the realities that they face in their classrooms. We 
also recognize that teachers should be content experts and should be treated as such. In this 
process, we also hope to expand the conversation about content quality so that it takes into 
consideration aspects of quality beyond mere accuracy of information that are essential for 
educational materials to actually be useful in authentic classrooms. The driving research question 
for this study is: How do teacher evaluations of textbook quality compare between copyright- 
restricted, open, and open/adapted textbooks? 

Toward this end, this study makes three comparisons between teacher evaluations of copyright- 
restricted, open, and open/adapted textbooks as follows: 1) open vs. copyright-restricted (O-C), 
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2) open/adapted vs. open (A-O), and 3) open/adapted vs. copyright-restricted (A-C). By 
comparing these evaluations, we hope to provide evidence to guide decision-making in the 
adoption, adaptation, or rejection of open textbooks for K-12 institutions. 

Context 

This study was conducted at a public research university in the northwestern United States. Some 
teachers may not have the expertise, time, or resources necessary to develop their own open 
textbooks due to local realities and lack of training (cf. Petrides, Jimes, Middleton-Detzner, 
Walling, & Weiss, 2011; Unwin et al., 2010), and professional development and engaging in 
professional learning communities with other educators toward this goal can help (DuFour, 2004; 
Kimmons, 2014; Wilson-Strydom, 2009). For these reasons, this study took place in conjunction 
with a three-day institute focused on helping participants create and adapt open textbooks. This 
approach has been found to be successful through previous research (Kimmons, 2014). The 
institute was announced via email to teachers throughout the target state, and applications were 
accepted from any practicing teacher in the state who taught in one of the targeted institute grade 
level and subject areas. These included: U.S. History (8 th grade), Algebra 2 (io-n th ), and all 
Middle and High School Sciences. In total, thirty (n = 30) teachers participated, serving as 
content creators and evaluators. 

During these institutes, participating teachers learned about OER-related topics like copyright, 
Creative Commons (n.d.), and public domain and were also given time and guidance to evaluate, 
adapt, and create assigned resources. As part of participation, the institute paid some teacher 
costs including lodging, professional development credits, and food for the duration of the 
institute and also offered to provide printed classroom sets of textbooks created through the 
institute for participants’ ongoing classroom use. Teachers applied to attend the institute by filling 
out a short survey wherein they documented their experience in the classroom and the subject 
area and grade levels wherein they would be interested and qualified for participating in the 
creation of an open textbook. Prior to attending this institute, many participants had very limited 
background on the concept of OER, and several participants arrived and stated that they did not 
fully understand what they would be doing in the institute but that it sounded interesting, 
revealing that participants were drawn to the idea of creating a textbook but were unclear how 
this would be possible within the allotted timeframe. 

Participating teachers were also encouraged to bring a copy of their current copyright-restricted 
textbooks that they use in their classroom so that they could be used for comparison and 
evaluation purposes. Early in the first day of the institute, participants were organized into 
subject area and grade level groups of between three and six people and were instructed to have a 
discussion about their existing copyright-restricted textbooks, taking notes on what their 
textbooks did well and what they did not do well. These conversations lasted around 45 minutes, 
and collaborative documents were created to guide ongoing conversations. Because participants 
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came from different schools, some had not been using the same textbooks and some had been 
using copyright-restricted e-books. Upon concluding these discussions, participants individually 
completed an evaluation of their copyright-restricted textbooks using the textbook evaluation 
survey form. Copyright-restricted textbooks included teacher-provided examples from prominent 
publishers such as Glencoe/McGraw-Hill, Houghton Mifflin Harcourt, Discovery Education, 
Prentice Hall, and others. 

Participants were then provided with a link to one or more online exemplary open textbooks for 
their grade level and subject area from either CK-12 or OpenStax. Participants were then given 45 
minutes to an hour to go through the contents of the open textbook as a group, adding to their 
previous collaborative document and taking notes on its strengths and limitations. Again, 
participants were instructed to individually evaluate this exemplary textbook utilizing the 
textbook evaluation survey. 

The institute progressed with lecture and discussion about OER, laying groundwork 
understanding on issues of copyright, public domain, Creative Commons, and so forth. 
Participants were then guided in a step-by-step manner in selecting an existing open textbook as a 
foundational text. These textbooks were then iteratively adapted to their needs by aligning them 
with standards, updating content, adding additional multimedia resources, and so forth. The 
entirety of the second day as well as significant portions of the first and third days were devoted to 
this process of adaptation. Near the end of the institute, participants shared their products with 
the larger group and went through troubleshooting processes to determine how they would share 
their created resources with their students and utilize them in their classrooms either as an 
electronic resource or as a printed textbook. When completed, all printed textbooks produced 
from this institute cost between $3 and $15 USD per copy without benefits of bulk or educational 
pricing. 

Upon completion of their open/adapted textbooks, participants were again directed to complete 
the textbook evaluation survey for the open/adapted textbook and also to complete an evaluation 
of the institute itself as a learning experience. Thus, data sources for this study consisted of survey 
results from three separate institutes with three data collection conditions for each institute, 
representing each textbook type: copyright-restricted (C), open (O), and open/adapted (A). 

Textbook Evaluation Survey 

Items in the textbook evaluation survey were constructed out of a previous research study by the 
author with a similar participant group, wherein participants identified potentials and barriers of 
OER adoption (Kimmons, in press). Participants were provided with the following instructions: 
“Please rate the textbook {textbook name here} on each of the following items.” Items included 
the following: 

1 . Accuracy: Accuracy of Content (e.g., accurate information) 
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2 . Aesthetics: Aesthetics (e.g., cover, color usage) 

3 . Alignment: Standards Alignment 

4 . Conciseness: Conciseness (e.g., unnecessary content is removed) 

5 . Formatting: Formatting of Text (e.g., font size, spacing) 

6. Media: Quality of Images and other Media 

7 . Readability: Readability of Content (e.g., grade level appropriateness) 

8. Resources: Connections to other Meaningful Resources (e.g., Internet resources) 

9 . Supplements: Supplementary Materials (e.g., teacher's guide, equation 
references) 

10 . Timeliness: Timeliness of Content (e.g., up-to-date information) 

Participants evaluated each of these items on a 5-point Likert scale representing the following 
values: Poor, Fair, Good, Very Good, and Excellent. 

Initially, the survey only included nine items, but after conducting the survey with the first group, 
it became clear that alignment should be included as an evaluation item, based upon feedback 
from participants. For this reason, results for the alignment item have fewer responses than the 
other items. All survey items were averaged together to create an overall evaluation of each 
textbook, and the evaluation survey was determined to be reliable with a very high Cronbach’s 
Alpha of .91 between the individual evaluation items. 

Delimitations 

One argument against this evaluation approach may be that because participants brought their 
current, copyright-restricted textbooks with them (some of which were several years old), it may 
not be fair to compare these to potentially newer open textbooks, because one might be older than 
the other. However, as discussed above, this is a reality in K-12 settings where the quality of 
materials is influenced by cost and affordability. If the goal of this study was to inform one-time 
adoption of such textbooks, then it might make sense to compare only new copyright-restricted 
textbooks with new open textbooks. However, our goal is to understand this phenomenon as it 
actually exists in schools, wherein new copyright-restricted materials are not a viable option in an 
on-going way (i.e. between adoption cycles). If a school has the funds to start a new cycle, then 
up-to-date texts of both types could be meaningfully compared, but once this adoption is made, a 
timer begins until another copyright-restricted adoption may be made again, and within this 
countdown, schools will perpetually face the real opportunity costs for having adopted copyright- 
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restricted textbooks in contrast to open textbooks that might be more regularly updated. Thus, 
given the realities of school adoptions, comparing OER with what teachers currently have instead 
of what they could have in an unrealistic alternate reality seems to make more experimental 
sense. This decision may influence results of the O-C and A-C comparisons, but not the A -0 
comparison. 

Furthermore, this study focuses upon a very specific set of open textbooks created by CK-12 and 
OpenStax for a limited number of subject areas and grade levels. As such, results should not be 
extrapolated to apply to all OER irrespective of source or subject. Rather, these specific sources, 
subject areas, and grade levels were selected specifically because they provided a number of 
widely-used, professionally-created resources that seemed to be promising grounds for exploring 
the issue of quality. 


Results 

Results of this study revealed differences in all three comparisons, as textbook evaluations 
descriptively improved on every item from copyright-restricted to open and again from open to 
open/adapted (cf. Fig. 1). 

Fig. 1. Graphical results of textbook evaluations across all three type conditions. 


■ copyright-restricted (C) ■ open (O) ■ open-adapted (A) 



Overall, the open textbooks were rated 22% higher than the copyright-restricted textbooks, the 
open/adapted textbooks were rated 16% higher than the open textbooks, and the open/adapted 
textbooks were rated 38% higher than the copyright-restricted textbooks (cf. Table 1). 
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Table l. Descriptive results of textbook evaluations for each condition. 


copyright-restricted 

(C) 

open (O) 

open/adapted (A) 


n 

Mean 

SD 

n 

Mean 

SD 

n 

Mean 

SD 

accuracy 

30 

.6 

.21 

28 

.68 

.19 

29 

•79 

.18 

aesthetics 

30 

48 

.21 

28 

•54 

.22 

29 

.82 

•13 

alignment 

19 

•32 

.26 

19 

.66 

.28 

20 

.89 

•17 

conciseness 

30 

• 3 i 

.22 

28 

.62 

.2 

29 

•79 

.18 

formatting 

30 

.61 

•23 

28 

.62 

.19 

29 

•83 

.18 

media 

30 

•53 

•24 

28 

•72 

.21 

29 

•83 

.18 

readability 

30 

•44 

•23 

28 

•7 

.16 

29 

•83 

•14 

resources 

30 

•25 

•25 

28 

•71 

•24 

29 

•85 

•17 

supplements 

30 

•4 

•27 

28 

•47 

•25 

29 

.62 

.26 

timeliness 

30 

•35 

.28 

28 

.78 

.16 

29 

•85 

•14 

overall (avg) 

30 

•43 

•13 

28 

•65 

.1 

29 

.81 

•13 


To determine whether evaluation differences were statistically significant for the sample size, a 
series of one-way analysis of variance (ANOVA) tests of textbook type (independent) effects on 
evaluation items (dependents) yielded significant differences in all factors (cf. Table 2). 
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Table 2. ANOVA results of textbook type group effects on individual evaluation items. 



Sum of Squares 

df 

Mean Square 

F 

accuracy 

•56 

2 

.28 

7-3 ** 

aesthetics 

1.9 

2 

•95 

26.04 *** 

alignment 

3.22 

2 

1.61 


conciseness 

3-56 

2 

1.78 

43.88 *** 

formatting 

•9 

2 

•45 

11.08 *** 

media 

1-4 

2 

•7 

15.75 *** 

readability 

2.28 

2 

1.14 

34.54 *** 

resources 

5-85 

2 

2.92 

58.53 *** 

supplements 

•74 

2 

•37 

5-59 ** 

timeliness 

4-35 

2 

2.17 

51-34 *** 

overall (avg) 

2.09 

2 

1.04 

68.91 *** 


** Denotes significance at thep < .01 level. 

*** Denotes significance at thep < .001 level. 

Given the significance of the overall tests, Bonferonni post hoc testing was utilized to determine 
differences between individual groups. The A-C comparison significantly differed in all factors, 
while the C -0 and A -0 comparisons differed on six of ten factors (cf. Table 3). 
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Table 3. Bonferonni post hoc results of textbook type group effects on evaluation items. 


open (O) vs. 
copyright-restricted 
(C) [O-C] 

open/adapted (A) vs. 
open (O) [A-O] 

open/adapted (A) vs. 
copyright-restricted 
(C) [A-C] 


Mean 

Difference 

Std. 

Error 

Mean 

Difference 

Std. 

Error 

Mean 

Difference 

Std. 

Error 

accuracy 

.08 

•05 

.11 

•05 

.19 ** 

•05 

aesthetics 

•05 

•05 

2g *** 

•05 

24 *** 

•05 

alignment 

*** 

.08 

.23* 

.08 

•57 *** 

.08 

conciseness 


•05 

.18** 

•05 

.48 *** 

•05 

formatting 

.01 

•05 

2^ *** 

•05 

22 *** 

•05 

media 

2 ** 

.06 

.1 

.06 

2 *** 

■05 

readability 

.25 *** 

•05 

.13* 

•05 

2^ *** 

•05 

resources 

.46 *** 

.06 

.15* 

.06 

^ *** 

.06 

supplements 

.07 

.07 

•15 

.07 

22 

.07 

timeliness 

42 *** 

•05 

.08 

•05 

r *** 

•0 

•05 

overall (avg) 

22 *** 

•03 

.16 *** 

■03 

2 rj *** 

•03 


* Denotes significance at thep < .05 level. 

**Denotes significance at thep < .01 level. 

*** Denotes significance at thep < .001 level. 

Qualitative feedback regarding the experience of adaptation was also gathered from participants 
in order to add depth to the quantitative results provided above. Some statements provided by the 
participants that illustrated their feelings upon completing the institute included the following: 

• “Outstanding ... extremely valuable. This [institute] has opened the door to a world of 
opportunity” (Pi). 
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• “This was an extremely useful experience. I can't wait to start using this open [textbook] 
in my classroom” (P2). 

• “This was AWESOME!! The product we created will be immensely useful and has given 
me a great tool for all of my students to use. The idea that all of the information is in one 
place will make my life a ton easier. I am ecstatic” (P3). 

• “So useful. My district is interested in moving this direction due to its cost-effectiveness. 
I was mostly concerned with losing out on the copywrite-restricted resources that I 
already use in my class, but now I see how I can use both” (P4). 

• “It's beyond easy” (P5). 

• “This [institute] was way more engaging than most classes I've taken for teaching. I also 
feel like it's the best product I've ever came out with” (P6). 

• “This ... was [an institute] that I will actually apply almost daily in my class” (P7). 

• “I am very motivated to use the [textbook] I created in my classroom next year and to 
share what I learned with my colleagues” (P8). 

• “Please do whatever you can to keep these [institutes] going!” (P9). 

• “This stuff will help teachers everywhere!!” (P10). 

Discussion and Implications 

This study provides two major findings for K-12 practitioners, researchers, and decision-makers. 
First, it is clear from the results that both the open textbooks and the open/adapted textbooks 
were evaluated by K-12 teachers as being superior to their copyright-restricted alternatives. 
Though part of this difference may be attributable to the relative ages of the resources and 
benefits emerging directly from cost that allows OER to be adopted outside of traditional 
textbooks cycles, results indicate that improved quality is also significant in other factors that 
have nothing to do with the age of the resource, including conciseness, readability, and additional 
resources. This means that the benefit of open textbooks extends beyond just cost and factors 
correlated to cost but seems also to be influenced by factors related to openness itself and the 
paradigmatic shift that open textbooks are permitting within schools, wherein teachers can take a 
more active role in resource evaluation, adoption, and modification. This means that though cost 
is certainly a major benefit of OER that influences other aspects of resource quality, such as 
timeliness, benefits of OER do not stop there. Rather, open textbooks are allowing for the creation 
of new resources that are better serving the needs and expectations of K-12 teachers. 
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The second major finding from this study is that though open textbooks may be simply adopted in 
a manner consonant with the copyright-restricted textbook paradigm (i.e. without modification or 
adaptation), the fact that these resources can be adapted, remixed, and modified means that they 
can be of even greater value to educators in addressing local contextual needs. It is noteworthy 
that though the O-C comparison was large and significant, the A -0 comparison was also 
significant and almost as large. This means that to get full use out of OER and to have the highest 
quality resources possible, K-12 teachers need time, training, and opportunities to adapt these 
resources to their particular needs. Thus, though a shift to open in the form of open textbook 
adoption may provide some benefits to K-12 teachers and institutions, the full benefit of a shift to 
open can only be achieved if open practices such as remixing and adaptation are allowed, 
encouraged, and supported (cf. Kimmons, in press). This result may also be applicable in higher 
education, because though cost may not be a major guide for OER adoption among faculty, the 
improved quality of adapted vs. merely adopted materials might be a benefit for educators at all 
level. 


Conclusion 

This study has sought to establish how the quality of open textbooks and open/adapted textbooks 
can compare to copyright-restricted alternatives. In this study, open textbooks included pre¬ 
made, openly-licensed textbooks made available through an online non-profit, such as CK-12 or 
OpenStax, while open/adapted textbooks represented teacher remixes of open textbooks. Results 
of the study indicate that in the subject areas and grade levels presented, open textbooks were of 
higher quality than copyright-restricted textbooks, and open/adapted textbooks were of higher 
quality than open textbooks. These results mark an early step in guiding K-12 institutions in a 
shift toward open adoption and adaptation and provides evidence that open alternatives can be 
viable. This study has also demonstrated how a number of K-12 OER quality assurance factors 
may be included in teacher-based evaluations that may improve traditional quality assurance 
processes beyond content-expert peer-review, thereby ensuring that quality becomes a more 
concrete measure and reflects actual usefulness for classroom teachers. 
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